Papers with annotation costs
Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarks (2023.findings-eacl)
Copied to clipboard
Pedro Rodriguez, Mahmoud Azab, Becka Silvert, Renato Sanchez, Linzy Labson, Hardik Shah, Seungwhan Moon
| Challenge: | Existing benchmarks for text-to-video retrieval are incomplete, resulting in false negatives . a recent state-of-the-art model gains 25% recall points, but this is not the case for TVR. |
| Approach: | They propose to retire video captioning datasets as TVR benchmarks . they propose to annotate and release additional caption-video pairs to mitigate this flaw . |
| Outcome: | The proposed method fails to accurately reflect reality, despite lack of purpose-built benchmarks. |
Autodive: An Integrated Onsite Scientific Literature Annotation Tool (2023.acl-demo)
Copied to clipboard
| Challenge: | Annotating scientific literature directly on PDF documents can greatly improve the labeling efficiency of scientists whose annotation costs are very high. |
| Approach: | They propose an integrated onsite scientific literature annotation tool for natural scientists and Natural Language Processing (NLP) researchers. |
| Outcome: | The proposed tool supports the whole lifecycle of corpus generation including i)project management, ii)resource management, and iv)ontology management, as well as manual annotation, onsite auto annotation, and vi)task statistic. |
Active DOP: A constituency treebank annotation tool with online learning (C18-2)
Copied to clipboard
| Challenge: | a new language-independent treebank annotation tool supports rich annotations with discontinuous constituents and function tags. |
| Approach: | They propose a language-independent treebank annotation tool supporting rich annotations with discontinuous constituents and function tags. |
| Outcome: | The proposed tool supports rich annotations with discontinuous constituents and function tags. |
Bag of Experts Architectures for Model Reuse in Conversational Language Understanding (N18-3)
Copied to clipboard
| Challenge: | Slot tagging is a key component of natural language understanding systems for personal digital assistants. |
| Approach: | They propose to use a bag of experts architecture to reuse domain data for slot tagging models. |
| Outcome: | Experiments with 10 domains show that the proposed models outperform baseline models by 5.06% and 12.16% when training with only 25% of the training data. |
DocumentNet: Bridging the Data Gap in Document Pre-training (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Document understanding tasks are a tedious task that requires extensive training and privacy constraints. |
| Approach: | They propose a method to collect weakly labeled data from the web to benefit VDER training . the collected dataset does not depend on specific document types or entity sets . |
| Outcome: | The proposed method does not depend on specific document types or entity sets, making it universally applicable to all VDER tasks. |
An Efficient Framework for Whole-Page Reranking via Single-Modal Supervision (2026.acl-industry)
Copied to clipboard
Zishuai Zhang, Sihao Yu, null Xiewenyi, Ying Nie, Junfeng Wang, Zhiming Zheng, Dawei Yin, Hainan Zhang
| Challenge: | Existing whole-page reranking methods require large-scale expert annotations to achieve high-quality results. |
| Approach: | They propose a whole-page reranking framework that converts single-modal rankers into page-level guidance by constructing budget-aware candidates for cross-modal annotations and distilling intra-modality preferences to align relevance scales across modalities. |
| Outcome: | The proposed framework reduces annotation costs by 70-90% while outperforming fully-annotated reranking baselines. |
Nuanced Toxicity Detection in Spanish: A New Corpus and Benchmark Study (2026.findings-eacl)
Copied to clipboard
Alba María Mármol-Romero, Robiert Sepúlveda-Torres, Estela Saquete, María-Teresa Martín-Valdivia, L. Alfonso Ureña
| Challenge: | Existing corpora for Spanish are under-resourced for toxic content detection . sarcasm, indirect aggression, irony, and other toxicity are not detected in English . |
| Approach: | They propose to extend the NECOS-TOX corpus to include 4,011 Spanish comments . each comment is annotated across three levels of toxicity, with substantial inter-annotator agreement . |
| Outcome: | The proposed model performs on par with larger models and is released publicly . the proposed model is based on a human-in-the-loop active learning strategy . |
FlexDoc: Parameterized Sampling for Diverse Multilingual Synthetic Documents for Training Document Understanding Models (2025.emnlp-industry)
Copied to clipboard
Karan Dua, Hitesh Laxmichand Patel, Puneet Mittal, Ranjeet Gupta, Amit Agarwal, Praneet Pabolu, Srikant Panda, Hansa Meghwani, Graham Horwood, Fahad Shah
| Challenge: | Document understanding models require large, diverse, and well-annotated datasets that can cost millions of dollars to collect and maintain. |
| Approach: | They propose a scalable synthetic data generation framework that combines Stochastic Schemas and Parameterized Sampling to produce realistic, multilingual semi-structured documents with rich annotations. |
| Outcome: | Experiments on key information extraction tasks show that the proposed framework improves the absolute F1 score by up to 11% while reducing annotation effort by over 90% compared to traditional hard-template methods. |
Annotations on a Budget: Leveraging Geo-Data Similarity to Balance Model Performance and Annotation Cost (2024.lrec-main)
Copied to clipboard
| Challenge: | Current foundation models have shown impressive performance across various tasks, but they are not effective for everyone due to the imbalanced geographical and economic representation of the data used in the training process. |
| Approach: | They propose to identify the data to be annotated to balance model performance and annotation costs by finding countries with visual similarity for the topics. |
| Outcome: | The proposed methods improve model performance and reduce annotation costs by using data from countries with higher visual similarity for these topics. |
Reassessing Active Learning Adoption in Contemporary NLP: A Community Survey (2026.eacl-long)
Copied to clipboard
| Challenge: | a longstanding strategy to reduce annotation costs is active learning . data annotation is expected to remain important and active learning to stay relevant . |
| Approach: | They conduct an online survey to assess the perceived relevance of data annotation and active learning . they propose a strategy to reduce annotation costs using active learning, an iterative process . |
| Outcome: | The proposed strategies reduce setup complexity and uncertainty cost while maintaining model performance. |
Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer in Prompt Tuning (2024.eacl-long)
Copied to clipboard
Kaige Xie, Tong Yu, Haoliang Wang, Junda Wu, Handong Zhao, Ruiyi Zhang, Kanak Mahadik, Ani Nenkova, Mark Riedl
| Challenge: | Existing prompt transfer techniques lack consideration for dialogue-specific information. |
| Approach: | They propose a method which leverages skeleton generation as extra supervision that functions as a medium connecting the distinct source and target task. |
| Outcome: | The proposed method significantly outperforms baselines on two dialogue summarization benchmarks. |
Language-to-Space Programming for Training-Free 3D Visual Grounding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for 3D visual grounding have been proposed, but they are limited by the scarcity of 3D vision-language datasets and the high cost of annotations. |
| Approach: | They propose a method for training-free 3D visual grounding that uses LLM-generated codes to analyze 3D spatial relations among objects. |
| Outcome: | The proposed method achieves 52.9% accuracy on the Nr3D benchmark and significantly reduces grounding time and token costs. |
MedDistant19: Towards an Accurate Benchmark for Broad-Coverage Biomedical Relation Extraction (2022.coling-1)
Copied to clipboard
| Challenge: | Relation extraction in the biomedical domain is challenging due to the lack of labeled data and high annotation costs. |
| Approach: | They propose to use distant supervision to pair knowledge graph relationships with raw texts to tackle the scarcity of annotated data and to validate their results. |
| Outcome: | The proposed benchmarks are more accurate and consistent with existing benchmarks and show that there is no train-test leakage. |
ELAD: Explanation-Guided Large Language Models Active Distillation (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are hindered by their memory inefficiency, computational demands, and the high costs of API inferences. |
| Approach: | They propose an Explanation-Guided LLMs Active Distillation framework that employs an active learning strategy to optimize the balance between annotation costs and model performance. |
| Outcome: | The proposed framework significantly improves the efficiency of LLMs knowledge distillation. |
ECC: An Emotion-Cause Conversation Dataset for Empathy Response (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing empathy dialogue datasets focus on emotion labels while cause annotations are added post hoc. |
| Approach: | They propose an emotion-cause conversation dataset with 2.4K dialogues that can be scalable . they use a framework that utilizes knowledge and large language models to automatically generate dialogues . |
| Outcome: | The proposed dataset can achieve comparable or even superior performance to existing empathy dialogue datasets. |
Self-Supervised Contrastive Learning for Efficient User Satisfaction Prediction in Conversational Agents (2021.naacl-main)
Copied to clipboard
| Challenge: | End-to-end deep learning methods that focus on user satisfaction are challenging due to the required annotation costs and turnaround times. |
| Approach: | They propose a self-supervised contrastive learning approach that leverages the pool of unlabeled data to learn user-agent interactions. |
| Outcome: | The proposed approach reduces the required number of annotations while improving generalization on unseen skills. |
Active Learning with Non-Uniform Costs for African Natural Language Processing (2026.findings-eacl)
Copied to clipboard
| Challenge: | Annotating datasets for African languages is challenging due to the continent's vast linguistic diversity, complicating development of NLP systems. |
| Approach: | They propose a cost-aware active learning method that integrates BatchBALD acquisition strategy with a 0-1 Knapsack optimization objective to select informative and budget-efficient samples. |
| Outcome: | The proposed method outperforms BALD, BatchBALD, and stochastic sampling variants across cost scenarios on the MasakhaNEWS multilingual news classification benchmark covering 11 African languages. |
iTAG: Inverse Design for Natural Text Generation with Accurate Causal Graph Annotations (2026.acl-long)
Copied to clipboard
| Challenge: | Lack of causally annotated text data for use as ground truth hinders causal discovery . early template-based generation methods sacrifice text naturalness in exchange for high annotation costs . |
| Approach: | They propose a method which performs real-world concept assignment to nodes before converting causal graphs into text. |
| Outcome: | The proposed method shows high annotation accuracy and naturalness across extensive tests. |
Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale (2024.emnlp-main)
Copied to clipboard
Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Chen, Xidong Wang, Zhenyang Cai, Ke Ji, Xiang Wan, Benyou Wang
| Challenge: | Multimodal large language models (MLLMs) lack visual knowledge in medical applications due to data privacy concerns and high annotation costs. |
| Approach: | They refined medical image-text pairs from PubMed and employed MLLMs (GPT-4V) to denoise and reformat the data. |
| Outcome: | The proposed model significantly improves the MMMU Health & Medicine track and shows that it can be used in multimodal scenarios. |
Assist Non-native Viewers: Multimodal Cross-Lingual Summarization for How2 Videos (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal summarization methods are limited to monolingual videos . a proposed task aims to generate cross-lingual summaries from multimodal inputs . |
| Approach: | They propose a task to generate cross-lingual summaries from multimodal inputs of videos . they propose fusion network that integrates multimodal and cross-linguistic information . |
| Outcome: | The proposed task outperforms existing methods on a reorganized How2 dataset on the reorganized How2 data set. |
Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation (2024.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that the model can be used to generate dialogues in new domains quickly. |
| Approach: | They propose to use LLMs to generate dialogue data to reduce dialogue collection and annotation costs. |
| Outcome: | The proposed model performs better than the baseline model trained on real data. |
A Semi-supervised Approach for De-identification of Swedish Clinical Text (2020.lrec-1)
Copied to clipboard
| Challenge: | An abundance of electronic health records (EHRs) is produced every day within healthcare. |
| Approach: | They propose a semi-supervised method for automatically creating high-quality training data for de-identification using annotated data for training and annotations that are costly in time and human resources. |
| Outcome: | The proposed method improves recall from 84.75% to 89.20% without sacrificing precision to the same extent, dropping from 95.73% to 94.20%. |
A Herd of Language Models Makes a Better Zero-shot Annotator for Clinical Named Entity Recognition (2026.findings-acl)
Copied to clipboard
| Challenge: | Clinical named entity recognition (NER) is a core task in clinical NLP. |
| Approach: | They propose a label-modeling method for M**ulti-LLM **A**nnotation using **R**epresentation learning to capture contextual similarity. |
| Outcome: | The proposed method improves the average F1 score by 8.6% over zero-shot baselines while reducing annotation costs. |
A Pilot Study on Dialogue-Level Dependency Parsing for Chinese (2023.findings-acl)
Copied to clipboard
| Challenge: | Dialogue-level dependency parsing has received insufficient attention, especially for Chinese. |
| Approach: | They propose a signal-based method to transform seen syntactic dependencies into unseen ones between elementary discourse units (EDUs) they apply single-view and multi-view data selection to access reliable pseudo-labeled instances. |
| Outcome: | The proposed method transforms seen syntactic dependencies into unseen ones between elementary discourse units (EDUs) the proposed method also provides reliable pseudo-labeled instances. |
Cold-start Active Learning through Self-supervised Language Modeling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Labeling data is a fundamental bottleneck in machine learning due to annotation cost and time. |
| Approach: | They propose a strategy that uses the pre-training loss to find examples that surprise the model and minimize labeling costs. |
| Outcome: | The proposed approach reduces labeling costs and costs by using pre-trained language models. |
Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning (2025.acl-long)
Copied to clipboard
Fangzhi Xu, Hang Yan, Chang Ma, Haiteng Zhao, Qiushi Sun, Kanzhi Cheng, Junxian He, Jun Liu, Zhiyong Wu
| Challenge: | Existing methods for enhancing LLM reasoning rely on supervisory signals . current methods rely heavily on outcome supervision and auxiliary reward models . |
| Approach: | They propose a gen-eralizable and purely unsupervised self-training framework to enhance LLM reasoning without supervision. |
| Outcome: | The proposed framework improves LLM reasoning without supervision without external supervision. |
Instruction Position Matters in Sequence Generation with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can perform conditional sequence generation tasks, such as translation or summarization, through instruction fine-tuning. |
| Approach: | They propose to shift the position of task instructions after the input sentences to enhance the model's instruction-following capability. |
| Outcome: | The proposed method outperforms traditional settings across various model scales (1B / 7B & 13B) and different sequence generation tasks (translation and summarization) without any additional data or annotation costs. |
UniSumm and SummZoo: Unified Model and Diverse Benchmark for Few-Shot Summarization (2023.acl-long)
Copied to clipboard
| Challenge: | a new benchmark summarization model is being developed to train few-shot summarizers . a large number of summarizing tasks are required to perform well in heterogeneous datasets. |
| Approach: | They propose a few-shot summarization model pre-trained with multiple summarizing tasks . they propose 'uniSumm' to be prefix-tuned to excel at any few-shot summarisation task . |
| Outcome: | The proposed model outperforms baseline models under automatic and human evaluations and achieves comparable results in human evaluation. |
Easy Samples Are All You Need: Self-Evolving LLMs via Data-Efficient Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that EasyRL consistently outperforms state-of-the-art baselines due to the substantial annotation cost and issues such as model collapse or reward hacking. |
| Approach: | They propose a supervised RL approach with a divide-and-conquer strategy that simulates the human cognitive acquisition curve using easy labeled data. |
| Outcome: | The proposed approach outperforms state-of-the-art models on mathematical and scientific benchmarks using only 10% of easy labeled data. |
Generating Structured Pseudo Labels for Noise-resistant Zero-shot Video Sentence Localization (2023.acl-long)
Copied to clipboard
| Challenge: | Existing zero-shot pipelines generate event proposals and then generate a pseudo query for each event proposal. |
| Approach: | They propose a Structure-based Pseudo Label generation (SPL) that generates free-form interpretable pseudo queries before constructing query-dependent event proposals. |
| Outcome: | The proposed method learns with only video data without any annotation . it generates free-form interpretable pseudo queries before constructing query-dependent event proposals . |
Structure-Aware Zero-Shot Relational Learning for Knowledge Graphs without External Knowledge (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Zero-shot Relational Learning depend on external knowledge, resulting in increased annotation costs and limited practical applicability. |
| Approach: | They propose a structure-aware paradigm that performs ZRL without external knowledge . it leverages intrinsic structural patterns in KGs to bridge semantic correlations for new relations with existing ones. |
| Outcome: | The proposed paradigm achieves 10.66% improvement in MRR while reducing annotation costs and enhancing practical applicability on three real-world benchmarks. |
Improving Task Diversity in Label Efficient Supervised Finetuning of LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities across domains . but, for challenging tasks, finetuning often requires substantial human annotations - a process that is time-consuming, labor-intensive, and expensive . |
| Approach: | They propose a method that leverages task-diversity as a principle for effective data selection. |
| Outcome: | The proposed method achieves better accuracy than training on the complete dataset (4% increase in MMLU score). |
Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation (2025.acl-long)
Copied to clipboard
Shuo Tang, Xianghe Pang, Zexi Liu, Bohan Tang, Rui Ye, Tian Jin, Xiaowen Dong, Yanfeng Wang, Siheng Chen
| Challenge: | Extensive experiments demonstrate that our framework effectively generates both general and domain-specific data. |
| Approach: | They propose a multi-agent simulator that automatically generates diverse text-based scenarios, capturing a wide range of real-world human needs. |
| Outcome: | Experiments show that the proposed model outperforms Meta’s Llama-3-8B-Instruct model on AlpacaEval 2 and Arena-Hard benchmarks with just 20K instruction-response pairs. |
How Real Are Synthetic Therapy Conversations? Evaluating Fidelity in Prolonged Exposure Dialogues (2025.findings-emnlp)
Copied to clipboard
Suhas Bn, Dominik O. Mattioli, Andrew M. Sherrill, Rosa I. Arriaga, Christopher Wiese, Saeed Abdullah
| Challenge: | Synthetic data adoption in healthcare is driven by privacy concerns, data access limitations, and high annotation costs. |
| Approach: | They compare real and synthetic PTSD therapy conversations using linguistic, structural, and protocol-specific metrics like turn-taking and treatment fidelity. |
| Outcome: | The proposed framework assesses clinical fidelity beyond surface fluency. |
Rebalancing Label Distribution While Eliminating Inherent Waiting Time in Multi Label Active Learning Applied to Transformers (2024.lrec-main)
Copied to clipboard
| Challenge: | Data annotation is a resourceintensive endeavor, necessitating human involvement and expertise. |
| Approach: | They propose to annotate instances to rebalance label distribution by judiciously selecting and limiting the data to be annotated. |
| Outcome: | The proposed method mitigates biases, improves model performance and reduces strategy-dependent disparities. |
SCOPE: Compress Mathematical Reasoning Steps for Efficient Automated Process Annotation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing process annotation approaches are computationally expensive. |
| Approach: | They propose a compression-based approach that transforms reasoning steps into code and normalizes them through Abstract Syntax Tree. |
| Outcome: | The proposed method outperforms existing methods on Best-of-N strategy and ProcessBench. |
Synthetic Data Generation for Training Diversified Commonsense Reasoning Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing Generative Commonsense Reasoning datasets are created using a small number of human annotators, covering only a narrow set of commonsense scenarios. |
| Approach: | They propose to use a synthetic dataset to train diverse commonsense generators. |
| Outcome: | The proposed model improves both generation diversity and quality compared with vanilla models and human-crafted datasets across different size Large Language Models (LLMs). |
Alignment Data Map for Efficient Preference Data Selection and Diagnosis (2026.findings-acl)
Copied to clipboard
| Challenge: | constructing high-quality preference datasets faces scalability challenges due to prohibitive cost and complexity of human annotation. |
| Approach: | They propose a tool to identify and select effective preference data by LLM-as-a-judge, explicit reward model, and reference-based approaches. |
| Outcome: | The proposed tool reduces annotation costs while preserving alignment effectiveness. |
Model in Distress: Sentiment Analysis on French Synthetic Social Media (2026.findings-acl)
Copied to clipboard
Pierre-Carl Langlais, Pavel Chizhov, Yannick Detrois, Carlos Rosas Hinostroza, Ivan P. Yamshchikov, Bastien Perroy
| Challenge: | Large companies and services handle a substantial volume of reviews and social media mentions. |
| Approach: | They propose a generalizable pipeline for automated customer feedback analysis using backtranslation and fine-tuned models to generate 1.7 million tweets from a small seed corpus, complemented by synthetic reasoning traces. |
| Outcome: | The proposed pipeline generates 1.7 million synthetic tweets from a small seed corpus, complemented by synthetic reasoning traces. |